Table of Contents
- 1. Introduction1
- 2. Importing the Required Libraries2
- 3. Loading the Data into Pandas Dataframes3
-
4. Data Preprocessing10
-
4.1 Data Cleaning11
- 4.1.1 Splitting "Policy_CombinedSingleLimit" column12
- 4.1.2 Changing Data-types 15
- 4.1.3 Dropping Unwanted Columns 18
- 4.1.4 Dealing with Duplicate Values 21
- 4.1.5 Feature Extraction From Dates Columns 24
- 4.1.6 Correlation Heatmap 27
- 4.2 Exploratory Data Anaysis (EDA) & Visualizations 30
- 4.1.7 Performing Train-Test Split 31
- 4.1.8 Handling Missing Values 33
- 4.1.9 Splitting Data Into Categorical and Numeric DataFrames 38
- 4.1.10 Standardization of Numeric DataFrame 42
- 4.1.11 Categorical Encoding 45
- 4.1.11.1 One-Hot Encoding46
- 4.1.11.2 Ordinal Encoding47
- 4.1.11.3 LabelEncoding48
- 4.1.11.4 Target Encoding49
- 4.1.12 Resetting the Index of Categorical & Numerical DataFrames 50
- 4.1.13 Combining Categorical & Numerical DataFrames 54
-
4.1 Data Cleaning11
- 5. Model Building 58
- 5.1 LazyPredict Classifier 59
- 5.2 Logistic Regression 60
- 5.3 Decision Tree Classifier 61
- 5.4 Random Forest Classifier 62
- 5.5 K-Nearest Neighbour 63
- 5.6 Support Vector Classifier 64
- 5.7 XGBoost 65
- 6. Ensemble Techniques 66
- 7. Learning Curves 69
- 8. K-Fold Cross Validation 70
- 9. Pattern Extraction 71
In [1]:
# Getting current date and time using now().
# importing datetime module for now()
import datetime
# using now() to get current time
current_time = datetime.datetime.now()
# Printing value of now.
print("Time now is:", current_time)
1. Introduction
The problem statement revolves around the detection of fraud in auto insurance claims, which is a critical issue faced by many general insurance companies. The fraudulent claims not only result in significant leakages for the insurer but also affect innocent people's lives. The insurance frauds can be classified based on their sources, including policyholders, intermediaries, and internal factors, and their nature, such as application, inflation, identity, fabrication, and staged accidents. Detecting and preventing frauds in auto insurance claims require a robust analytical and modelling framework that can predict the likelihood of fraud before processing the claims. The framework should also be capable of identifying the hidden patterns in the data that lead to fraudulent claims.
We are expected to perform exploratory data analysis, report the results from learning curves, build an analytical framework to predict fraud, and extract the top 20 patterns for fraudulent claims using decision tree algorithms only. This analytical and modelling framework can benefit not only the insurance companies but also the regulatory bodies and law enforcement agencies.
Objectives :
- To perform exploratory Data Analysis using visualizations
- To report the results/observations from learning curves
- To build the analytical framework to predict if a claim is fraudulent or not
- Finding Missing Data
- To extract the top 20 patterns for fraudulent claims using Decision Tree Algorithm
In [2]:
import os # Provides functions for creating and removing a directory (folder), fetching its contents, changing and identifying the current directory, etc.
import pandas as pd # To perform Operations on DataFrames.
import numpy as np # Perform a number of mathematical operations on arrays such as statistical, and algebraic.
import matplotlib.pyplot as plt
import missingno as msno
In [3]:
import pandas as pd
pd.set_option('display.max_rows', 500)
pd.set_option('display.max_columns', 500)
pd.set_option('display.width', 1000)
3. Loading the Data into Pandas Dataframes
We are provided with multiple CSV files, which are to be merged and used for Model Building.
Attributes Information Provided:¶
Demographics Data:
- CustomerID : Customer ID
- InsuredAge : age
- InsuredZipCode : Zip Code
- InsuredGender : Gender - the missing value is represented as “NA”
- InsuredEducationLevel : Education
- InsuredOccupation : Occupation
- InsuredHobbies : Hobbies
- CapitalGains : Capital gains(Financial Status)
- CapitalLoss : capital loss(Financial Status)
- Country : Country
Policy Information:
- CustomerID : Customer ID
- CustomerLoyaltyPeriod : Duration of customer relationship
- InsurancePolicyNumber : policy number
- DateOfPolicyCoverage : policy commencement date
- InsurancePolicyState : Policy location (State)
- Policy_CombinedSingleLimit : Split Limit and Combined Single Limit
- Policy_Deductible : Deductible amount
- PolicyAnnualPremium : Annual Premium – the missing value is represented as “-1”
- UmbrellaLimit : Umbrella Limit amount
- InsuredRelationship : Realtionship
Claim Information:
- CustomerID : Customer ID
- DateOfIncident : Date of incident
- TypeOfIncident : Type of incident
- TypeOfCollission : Type of Collision - “?” is the missing value
- SeverityOfIncident : Collision severity
- AuthoritiesContacted : Which authorities are contacted
- IncidentState : Incident location (State)
- IncidentCity : Incident location (City)
- IncidentAddress : Incident location (address)
- IncidentTime : time of incident – Hour of the day - the missing value is represented as “-5
- NumberOfVehicles : Number of vehicles involved
- PropertyDamage : If property damage is there - “?” is the missing value
- BodilyInjuries : Number of bodily injuries
- Witnesses : Number of witnesses - missing value is represented as "MISSINGVALUE"
- PoliceReport : If police report available - “?” is the missing value
- AmountOfTotalClaim : Total claim amount - the missing value is represented as “MISSEDDATA”
- AmountOfInjuryClaim : Claim for injury
- AmountOfPropertyClaim : claim for property damage
- AmountOfVehicleDamage : claim for vehicle damage
Data of Vehicle:
- CustomerID : Customer ID
- VehicleAttribute : Service signed for
- VehicleAttributeDetails : Value of the vehicle attribute - the missing value is represented as “???”
Fraud Data :
- CustomerID : Customer ID
- ReportedFraud : Fraud or not – Target
3.1 Loading Train Data
We are provided with 5 different files for Training Data.
Files are:
- Train_Demographics.csv
- Train_Policy.csv
- Train_Claim.csv
- Train_Vehicle.csv
- Traindata_with_Target.csv
Note:
- The data provided, comes with special missing value notations as metioned in the attribute description.
In [4]:
train_demographic = pd.read_csv("../Train Data/Train_Demographics.csv",na_values=['NA'])
train_policy = pd.read_csv("../Train Data/Train_Policy.csv",na_values=['NA', '-1', 'MISSINGVAL'])
train_claim = pd.read_csv("../Train Data/Train_Claim.csv",na_values=['?', '-5', 'MISSINGVALUE', 'MISSEDDATA'])
train_vehicle = pd.read_csv("../Train Data/Train_Vehicle.csv" ,na_values=['???'])
train_target = pd.read_csv("../Train Data/Traindata_with_Target.csv")
In [5]:
# print the shapes of the dataframes
print('Shape of train_demographic:', train_demographic.shape)
print('Shape of train_policy:', train_policy.shape)
print('Shape of train_claim:', train_claim.shape)
print('Shape of train_vehicle:', train_vehicle.shape)
print('Shape of train_target:', train_target.shape)
In [6]:
# print the shapes of the dataframes
print('columns of train_demographic:', train_demographic.columns)
print("♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦")
print('columns of train_policy:', train_policy.columns)
print("♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦")
print('columns of train_claim:', train_claim.columns)
print("♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦")
print('columns of train_vehicle:', train_vehicle.columns)
print("♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦")
print('columns of train_target:', train_target.columns)
3.1.1 Reshaping the "train_vehicle" DataFrame
In this datarame for every "CustomerID" there are four different values in the subsequent columns. We need to transform the unique values in "VehicleAttribute" column to individual columns and assign their respective value from "VehicleAttributeDetails" column.
In [7]:
train_vehicle.head()
Out[7]:
In [8]:
# pivot the dataframe to get unique values of VehicleAttribute as columns and VehicleAttributeDetails as the values
train_vehicle = train_vehicle.pivot_table(index='CustomerID', columns='VehicleAttribute', values='VehicleAttributeDetails', aggfunc='first')
train_vehicle.head()
Out[8]:
In [9]:
# removing the VehicleAttribute as index
train_vehicle = train_vehicle.rename_axis(None, axis=1)
# reset the index to convert the pivot table to a regular dataframe
train_vehicle = train_vehicle.reset_index()
# print the resulting head of the dataframe and its shape
display(train_vehicle.head(10))
print("\n")
print('Shape of train_vehicle:', train_vehicle.shape)
In [ ]:
In [10]:
# print the shapes of the dataframes
print('Shape of train_demographic:', train_demographic.shape)
print('Shape of train_policy:', train_policy.shape)
print('Shape of train_claim:', train_claim.shape)
print('Shape of train_vehicle:', train_vehicle.shape)
print('Shape of train_target:', train_target.shape)
In [11]:
# merge the dataframes based on the CustomerID column
merged_df = pd.merge(train_demographic, train_policy, on='CustomerID')
merged_df = pd.merge(merged_df, train_claim, on='CustomerID')
merged_df = pd.merge(merged_df, train_vehicle, on='CustomerID')
train_df = pd.merge(merged_df, train_target, on='CustomerID')
display(train_df.head(10))
print("\n")
print('Shape of train_df:', train_df.shape)
In [ ]:
In [12]:
test_demographic = pd.read_csv("../Test Data/Test_Demographics.csv",na_values=['NA'])
test_policy = pd.read_csv("../Test Data/Test_Policy.csv",na_values=['NA', '-1', 'MISSINGVAL'])
test_claim = pd.read_csv("../Test Data/Test_Claim.csv",na_values=['?', '-5', 'MISSINGVALUE', 'MISSEDDATA'])
test_vehicle = pd.read_csv("../Test Data/Test_Vehicle.csv" ,na_values=['???'])
test_target = pd.read_csv("../Test Data/Test.csv")
In [13]:
# print the shapes of the dataframes
print('Shape of test_demographic:', test_demographic.shape)
print('Shape of test_policy:', test_policy.shape)
print('Shape of test_claim:', test_claim.shape)
print('Shape of test_vehicle:', test_vehicle.shape)
print('Shape of test_target:', test_target.shape)
In [14]:
# print the shapes of the dataframes
print('columns of test_demographic:', test_demographic.columns)
print("♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦")
print('columns of test_policy:', test_policy.columns)
print("♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦")
print('columns of test_claim:', test_claim.columns)
print("♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦")
print('columns of test_vehicle:', test_vehicle.columns)
print("♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦")
print('columns of test_target:', test_target.columns)
3.2.1 Reshaping the "test_vehicle" DataFrame
In this datarame for every "CustomerID" there are four different values in the subsequent columns. We need to transform the unique values in "VehicleAttribute" column to individual columns and assign their respective value from "VehicleAttributeDetails" column.
In [15]:
# pivot the dataframe to get unique values of VehicleAttribute as columns and VehicleAttributeDetails as the values
test_vehicle = test_vehicle.pivot_table(index='CustomerID', columns='VehicleAttribute', values='VehicleAttributeDetails', aggfunc='first')
test_vehicle.head()
Out[15]:
In [16]:
# removing the VehicleAttribute as index
test_vehicle = test_vehicle.rename_axis(None, axis=1)
# reset the index to convert the pivot table to a regular dataframe
test_vehicle = test_vehicle.reset_index()
# print the resulting head of the dataframe and its shape
display(test_vehicle.head(10))
print("\n")
print('Shape of train_vehicle:', test_vehicle.shape)
In [17]:
# print the shapes of the dataframes
print('Shape of test_demographic:', test_demographic.shape)
print('Shape of test_policy:', test_policy.shape)
print('Shape of test_claim:', test_claim.shape)
print('Shape of test_vehicle:', test_vehicle.shape)
print('Shape of test_target:', test_target.shape)
In [18]:
# merge the dataframes based on the CustomerID column
merged_df = pd.merge(test_demographic, test_policy, on='CustomerID')
merged_df = pd.merge(merged_df, test_claim, on='CustomerID')
merged_df = pd.merge(merged_df, test_vehicle, on='CustomerID')
test_df = pd.merge(merged_df, test_target, on='CustomerID')
display(test_df.head(10))
print("\n")
print('Shape of test_df:', test_df.shape)
Steps for Data Pre-Processing:
- Decide numerical and categorical columns and change their respective data types.
- Check for missing values and duplicate values in the data.
- Check for cardinality in the columns.
- Decide X (Independent attributes or predictors) & y (Dependent variable).
- Do the Train-Test split.
a. train_test_split b. **Outcome:** * X_train, X_test, y_train, y_test - Separate categorical columns & numerical columns (X_train, X_test)
a. **Outcome:** * X_train_cat, X_train_num, X_test_cat, X_test_num - For Categorical columns:
a. Impute missing values (Mode) * Fit on X_train_cat * Transform on X_train_cat & X_test_cat b. Label Encoding * Only for columns which have ordinality or the target column. c. One-hot Encoding * Only for columns with no ordinality or the target column. * Better at handling errors, or new values generated when compared to get_dummies. * Can do fit_transform directly. - For Numerical columns:
a. Impute missing values (Mean, Median) * Fit on X_train_num * Transform on X_train_num & X_test_num b. Standardization: * Fit on X_train_num * Transform on X_train_num & X_test_num c. Normalization: * Fit on X_train_num * Transform on X_train_num & X_test_num - Reset the Index.
- Combine numerical data & categorical data.
4.1 Data Cleaning
We do the following steps in this part.
Steps:
- Splitting the "Policy_CombinedSingleLimit" column
- Dropping columns which contribute in poor performance of the model.
- Checking weather there are any duplicate records in the data, if so remove them.
- Handling missing data present in the data.
Note:
- These steps are to be done for both, Train Data & Test Data
In [19]:
display(train_df.head())
In [20]:
# create two new columns by splitting the values column
train_df[['SplitLimit', 'CombinedSingleLimit']] = train_df['Policy_CombinedSingleLimit'].str.split('/', expand=True)
# convert the columns to the appropriate data types if necessary
train_df['SplitLimit'] = train_df['SplitLimit'].astype(int)
train_df['CombinedSingleLimit'] = train_df['CombinedSingleLimit'].astype(int)
# dropping the original column
train_df.drop("Policy_CombinedSingleLimit", axis=1, inplace=True)
In [21]:
display(train_df.head())
In [22]:
print('Shape of train_df:', train_df.shape)
In [23]:
display(test_df.head())
In [24]:
# create two new columns by splitting the values column
test_df[['SplitLimit', 'CombinedSingleLimit']] = test_df['Policy_CombinedSingleLimit'].str.split('/', expand=True)
# convert the columns to the appropriate data types if necessary
test_df['SplitLimit'] = test_df['SplitLimit'].astype(int)
test_df['CombinedSingleLimit'] = test_df['CombinedSingleLimit'].astype(int)
# dropping the original column
test_df.drop("Policy_CombinedSingleLimit", axis=1, inplace=True)
In [25]:
display(test_df.head())
In [26]:
print('Shape of test_df:', test_df.shape)
4.1.2 Changing Data-types
In this step, we change the columns to their appropriate data-types as mentioned below.
Columns and their data-type:
- CustomerID: Category
- InsuredAge: int
- InsuredZipCode: Category
- InsuredGender: Category
- InsuredEducationLevel: Category
- InsuredOccupation:Category
- InsuredHobbies: Category
- CapitalGains: int
- CapitalLoss: int
- Country: Category
- InsurancePolicyNumber: Category
- CustomerLoyaltyPeriod: int
- DateOfPolicyCoverage: date
- InsurancePolicyState: Category
- Policy_Deductible: int
- PolicyAnnualPremium: float
- UmbrellaLimit: int
- InsuredRelationship: Category
- DateOfIncident: date
- TypeOfIncident: Category
- TypeOfCollission: Category
- SeverityOfIncident: Category
- AuthoritiesContacted: Category
- IncidentState: Category
- IncidentCity: Category
- IncidentAddress: Category
- IncidentTime: float
- NumberOfVehicles: int
- PropertyDamage: Category
- BodilyInjuries: int
- Witnesses: int
- PoliceReport: Category
- AmountOfTotalClaim: float
- AmountOfInjuryClaim: int
- AmountOfPropertyClaim: int
- AmountOfVehicleDamage: int
- VehicleID: Category
- VehicleMake: Category
- VehicleModel: Category
- VehicleYOM: date
- ReportedFraud: Category
- SplitLimit: int
- CombinedSingleLimit: int
In [27]:
train_df.dtypes
Out[27]:
In [28]:
# function to convert columns of a dataframe into metioned data-type
def convert_columns_types_to_category(DataFrame, cols=None, col_type = None):
display('### Before conversion: ###', DataFrame.dtypes) # Checking the Data Types of columns before converting
DataFrame[cols] = DataFrame[cols].astype(col_type) # Changing the Data Types using astype() function
display('### After conversion: ###', DataFrame.dtypes) # Checking the Data Types of columns before converting
return DataFrame
In [29]:
# categorical converrsion
# getting categoric columns in ca_cols variable
cat_cols = ['CustomerID', 'InsuredZipCode', 'InsuredGender', 'InsuredEducationLevel',
'InsuredOccupation', 'InsuredHobbies', 'Country', 'InsurancePolicyNumber',
'InsurancePolicyState', 'InsuredRelationship', 'TypeOfIncident', 'TypeOfCollission',
'SeverityOfIncident', 'AuthoritiesContacted', 'IncidentState', 'IncidentCity',
'IncidentAddress', 'PropertyDamage', 'PoliceReport', 'VehicleID',
'VehicleMake', 'VehicleModel', 'ReportedFraud']
# calling the convert_columns_types_to_category() function defined above
train_df = convert_columns_types_to_category(train_df, cols=cat_cols, col_type = 'category')
In [30]:
# datetime converrsion
# usning the pandas to_datetime() function we conver the columns into date-time format
train_df['DateOfPolicyCoverage'] = pd.to_datetime(train_df['DateOfPolicyCoverage'], format='%d-%m-%Y')
train_df['DateOfIncident'] = pd.to_datetime(train_df['DateOfIncident'], format='%d-%m-%Y')
train_df['VehicleYOM'] = pd.to_datetime(train_df['VehicleYOM'], format='%Y')
In [31]:
train_df.dtypes
Out[31]:
In [32]:
test_df.dtypes
Out[32]:
In [33]:
# categorical converrsion
# getting categoric columns in ca_cols variable
cat_cols = ['CustomerID', 'InsuredZipCode', 'InsuredGender', 'InsuredEducationLevel',
'InsuredOccupation', 'InsuredHobbies', 'Country', 'InsurancePolicyNumber',
'InsurancePolicyState', 'InsuredRelationship', 'TypeOfIncident', 'TypeOfCollission',
'SeverityOfIncident', 'AuthoritiesContacted', 'IncidentState', 'IncidentCity',
'IncidentAddress', 'PropertyDamage', 'PoliceReport', 'VehicleID',
'VehicleMake', 'VehicleModel']
# calling the convert_columns_types_to_category() function defined above
test_df = convert_columns_types_to_category(test_df, cols=cat_cols, col_type = 'category')
In [34]:
# datetime converrsion
# usning the pandas to_datetime() function we conver the columns into date-time format
test_df['DateOfPolicyCoverage'] = pd.to_datetime(test_df['DateOfPolicyCoverage'], format='%Y-%m-%d')
test_df['DateOfIncident'] = pd.to_datetime(test_df['DateOfIncident'], format='%Y-%m-%d')
test_df['VehicleYOM'] = pd.to_datetime(test_df['VehicleYOM'], format='%Y')
In [35]:
test_df.dtypes
Out[35]:
4.1.3 Dropping Unwanted Columns
By looking at the data we can determine columns which can be dropped before even building models.
Such Columns are:
- CustomerID: The column contains the Customer ID's which is unique to every customer, so it does'nt make any contribution in performance of the model, rather it increases the dimentionaly of the data when it is encoded and thus arise the problem of "curse of dimentionality".
- Country: This column contains only one value across all the records. So it makes next to none contribution in model's performance.
- InsurancePolicyNumber: Same like the CustomerID, this column also contains the unique ID's of policy numbers of the customers, and fo the same reasons raised above we can safely drop this column too.
- VehicleID: for the same reasons mentioned above
In [36]:
# Function to Drop Unccessary Columns in a Data Frame
def drop_unnecessary_columns(DataFrame, cols=None):
display('Before Dropping : ', DataFrame.columns) # Printing columns names before dropping selected columns
print("♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦")
DataFrame.drop(columns = cols, axis = 1, inplace = True) # Dropping the columns
print("♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦-♦")
display('After Dropping : ',DataFrame.columns) # Printing columns names after dropping selected columns
return DataFrame
In [37]:
# storing the columns which are to be dropped in drop_cols
drop_cols = ['CustomerID', 'Country', 'InsurancePolicyNumber', 'VehicleID']
# calling the function defined above to drop the columns
train_df = drop_unnecessary_columns(train_df, cols=drop_cols)
In [38]:
# storing the columns which are to be dropped in drop_cols
drop_cols = ['CustomerID', 'Country', 'InsurancePolicyNumber', 'VehicleID']
# calling the function defined above to drop the columns
test_df = drop_unnecessary_columns(test_df, cols=drop_cols)
In [39]:
# this function checks for duplicate records in the dataframe
def check_duplicates(df):
# Check for duplicate records
duplicate_rows = df[df.duplicated()]
# Check if there are any duplicate records
if duplicate_rows.empty:
print("There are no duplicate records.")
else:
print("Duplicate records:")
print(duplicate_rows)
In [40]:
check_duplicates(train_df)
In [41]:
check_duplicates(test_df)
In [42]:
# Derive Age of Vehicle on the day of Incident
train_df['VehicleAge'] = (train_df['DateOfIncident'] - train_df['VehicleYOM']).dt.days
# Derive Day of the Week of Incident
train_df['DayOfWeek'] = train_df['DateOfIncident'].dt.day_name()
# Derive Month of Incident
train_df['MonthOfIncident'] = train_df['DateOfIncident'].dt.month_name()
# Derive Time between Policy Coverage and Incident
train_df['TimeBetweenCoverageAndIncident'] = (train_df['DateOfIncident'] - train_df['DateOfPolicyCoverage']).dt.days
In [43]:
train_df.shape
Out[43]:
In [44]:
train_df.drop(['DateOfIncident', 'VehicleYOM', 'DateOfPolicyCoverage'], axis=1, inplace=True)
In [45]:
train_df.head()
Out[45]:
In [46]:
# Change the dtype of 'DayOfWeek', 'MonthOfIncident' column from object to category
train_df['DayOfWeek'] = train_df['DayOfWeek'].astype('category')
train_df['MonthOfIncident'] = train_df['MonthOfIncident'].astype('category')
In [47]:
train_df.head()
Out[47]:
In [48]:
train_df.dtypes
Out[48]:
In [49]:
# Derive Age of Vehicle on the day of Incident
test_df['VehicleAge'] = (test_df['DateOfIncident'] - test_df['VehicleYOM']).dt.days
# Derive Day of the Week of Incident
test_df['DayOfWeek'] = test_df['DateOfIncident'].dt.day_name()
# Derive Month of Incident
test_df['MonthOfIncident'] = test_df['DateOfIncident'].dt.month_name()
# Derive Time between Policy Coverage and Incident
test_df['TimeBetweenCoverageAndIncident'] = (test_df['DateOfIncident'] - test_df['DateOfPolicyCoverage']).dt.days
In [50]:
test_df.drop(['DateOfIncident', 'VehicleYOM', 'DateOfPolicyCoverage'], axis=1, inplace=True)
In [51]:
# Change the dtype of 'DayOfWeek', 'MonthOfIncident' column from object to category
test_df['DayOfWeek'] = test_df['DayOfWeek'].astype('category')
test_df['MonthOfIncident'] = test_df['MonthOfIncident'].astype('category')
In [52]:
test_df.head()
Out[52]:
In [53]:
import seaborn as sns
In [54]:
# Visualizing the correlation matrix by plotting heat map.
plt.style.use('seaborn-pastel')
upper_triangle = np.triu(train_df.corr())
plt.figure(figsize=(25,25))
sns.heatmap(train_df.corr(), vmin=-1, vmax=1, annot=True, square=True, fmt='0.2f',
annot_kws={'size':16}, cmap="plasma", mask=upper_triangle)
plt.xticks(fontsize=20)
plt.yticks(fontsize=20)
plt.show()
This heatmap shows the correlation matrix by visualizing the data. we can observe the relation between one feature to other.
This heatmap contains both positive and negative correlation.
We can observe the most of the columns are highly correlated with each other which leads to the multicollinearity problem.
In [55]:
# Visualizing the correlation matrix by plotting heat map.
plt.style.use('seaborn-pastel')
upper_triangle = np.triu(test_df.corr())
plt.figure(figsize=(25,25))
sns.heatmap(test_df.corr(), vmin=-1, vmax=1, annot=True, square=True, fmt='0.2f',
annot_kws={'size':16}, cmap="plasma", mask=upper_triangle)
plt.xticks(fontsize=20)
plt.yticks(fontsize=20)
plt.show()
Missing Values¶
In [56]:
msno.matrix(train_df)
plt.figure(figsize = (15,9))
plt.show()
In [57]:
msno.matrix(test_df)
plt.figure(figsize = (15,9))
plt.show()
Findings:
- There are large chunks of missing records in our data, as we can see in the graph above. Using mean, median & mode to impute the missing data would cause bias towards those classes. So its better to implement some advance techniques to impute the missing data.
Finding Potential Outliers¶
In [58]:
columns= ['InsuredAge',
'CapitalGains',
'CapitalLoss',
'CustomerLoyaltyPeriod',
'Policy_Deductible',
'PolicyAnnualPremium',
'UmbrellaLimit',
'IncidentTime',
'NumberOfVehicles',
'BodilyInjuries',
'Witnesses',
'AmountOfTotalClaim',
'AmountOfInjuryClaim',
'AmountOfPropertyClaim',
'AmountOfVehicleDamage',
'SplitLimit',
'CombinedSingleLimit',
'VehicleAge']
def plot_boxplot_side_by_side(df1, df2, columns):
# Create two subplots
fig, axs = plt.subplots(nrows=len(columns), ncols=2, figsize=(15, 5*len(columns)))
for i, col in enumerate(columns):
# Draw the first plot on the left
sns.boxplot(x=df1[col], ax=axs[i, 0],color="teal")
axs[i, 0].set_xlabel(col + " (Train)")
# Draw the second plot on the right
sns.boxplot(x=df2[col], ax=axs[i, 1],color="indigo")
axs[i, 1].set_xlabel(col + " (Test)")
# Show the plot
plt.show()
plot_boxplot_side_by_side(train_df, test_df, columns)
Findings:
- The following columns could contain potential outliers
- InsuredAge
- CustomerLoyaltyPeriod
- PolicyAnnualPremium
- UmbrellaLimit
- AmountOfTotalClaim
- AmpountOfInjuryClaim
- AmountOfPropertClaim
- AmountOfVehicleDamage
- We dont do anything to them as of now, as we cant say for sure they effect our model.
Checking the spread of Numeric Data¶
In [59]:
# columns= ['InsuredAge',
# 'CapitalGains',
# 'CapitalLoss',
# 'CustomerLoyaltyPeriod',
# 'Policy_Deductible',
# 'PolicyAnnualPremium',
# 'UmbrellaLimit',
# 'IncidentTime',
# 'NumberOfVehicles',
# 'BodilyInjuries',
# 'Witnesses',
# 'AmountOfTotalClaim',
# 'AmountOfInjuryClaim',
# 'AmountOfPropertyClaim',
# 'AmountOfVehicleDamage',
# 'SplitLimit',
# 'CombinedSingleLimit',
# 'VehicleAge']
# def plot_kdeplot_side_by_side(df1, df2, columns):
# # Create two subplots
# fig, axs = plt.subplots(nrows=len(columns), ncols=2, figsize=(15, 5*len(columns)))
# for i, col in enumerate(columns):
# # Draw the first plot on the left
# sns.kdeplot(df1[col], ax=axs[i, 0], fill=True, color='teal')
# axs[i, 0].set_xlabel(col + " (Train)")
# # Draw the second plot on the right
# sns.kdeplot(df2[col], ax=axs[i, 1], fill=True, color='indigo')
# axs[i, 1].set_xlabel(col + " (Test)")
# # Show the plot
# plt.show()
# plot_kdeplot_side_by_side(train_df, test_df, columns)
In [60]:
# Checking the skewness in our DataFrame
train_df.skew().sort_values()
Out[60]:
Findings:
- We see there is skewness in the data, so we need to normalize our data using StandardScaler from sklearn.
Distribution of individual classes in our DataFrame¶
In [61]:
def generate_pie(df):
plt.figure(figsize=(10,5))
plt.pie(df.value_counts(), labels=df.value_counts().index, autopct='%1.2f%%',shadow=True, explode=(0,.2), colors=['#006400' ,'#B22222'])
plt.legend(prop={'size':9})
plt.axis('equal')
return plt.show()
generate_pie(train_df.ReportedFraud)
From the plot we can observe that the count of "N" is high compared to "Y". Here most of the insurance claims have not reported as fradulent, which is usually the case.
Since it is our target column, it indicates the class imbalance issue. We will balance the data using oversampling method in later part.
In [62]:
# getting categoric columns in ca_cols variable
cat_cols = ['InsuredGender', 'InsuredEducationLevel',
'InsurancePolicyState', 'InsuredRelationship', 'TypeOfIncident',
'TypeOfCollission', 'SeverityOfIncident', 'AuthoritiesContacted',
'PropertyDamage', 'PoliceReport',
'MonthOfIncident']
def generate_pie(df):
plt.figure(figsize=(8,3))
plt.pie(df.value_counts(), labels=df.value_counts().index, autopct='%1.2f%%',shadow=True)
plt.legend(prop={'size':9})
plt.axis('equal')
return plt.show()
for col in train_df[cat_cols]:
print(f"Pie plot for the column:", col)
generate_pie(train_df[col])
Findings:
- InsuredGender: We see there are more females who are insured than males
- InsuredEducationLevel: As the plots projects, the people with 'JD' degrees are the most in numbeers to get insured and people who has college degrees are the least in numbers.
- InsurancePolicyState: "State3" has the highest population with people havings inurance and "State2" being the least.
- InsuredRelationship: From the graph we can derive that, 'own-child' was the most opted to be the nominee, followed by "not-in-family" in the second place and "unmaried" opted the last.
- TypeOfIncident: "Multi-vehicle Collision" is the most frequent incident wheres the "Parked Car" being the least.
- TypeOfCollision: "Rear Collision" is the most comman type of collision that happens whereas "Front Collision" being the least but by only 6%.
- SeverityOfIncident: Majority of the damage that happens to the vehicle are "Minor Damage", while "Trivial Damage" is the least.
- AuthoritiesContacted: "Police" was the most contacted after an incident, although therer are some cases where none of the authorities were contacted.
- PropertyDamage: Most of the time there was no property damage, although 47.29% times there was some property damage caused.
- PoliceReport: 52% of the time police report was not available.
- MonthOfIncident: 51.31% of the incidents happened in January, 47.87% in February and only 0.82% in March.
In [63]:
# getting categoric columns in ca_cols variable
cat_cols = ['InsuredOccupation', 'InsuredHobbies', 'IncidentState', 'IncidentCity', 'VehicleMake', 'DayOfWeek', 'MonthOfIncident']
def plot_countplots(df, columns):
# Calculate the number of rows and columns needed for the plot
n_cols = 2
n_rows = (len(columns) + 1) // 2
# Create the subplot grid
fig, axs = plt.subplots(nrows=n_rows, ncols=n_cols, figsize=(12, 4*n_rows))
plt.xticks(rotation=90)
# Flatten the subplot grid
axs = axs.flatten()
# Loop over the columns and plot a countplot for each one
for i, col in enumerate(columns):
sns.countplot(x=col, data=df, ax=axs[i],palette="gnuplot", )
axs[i].set_xticklabels(axs[i].get_xticklabels(), rotation=90)
# Remove any unused subplots
if len(columns) < n_rows * n_cols:
for i in range(len(columns), n_rows * n_cols):
fig.delaxes(axs[i])
# Display the plot
plt.tight_layout()
plt.show()
plot_countplots(train_df, cat_cols)
Finding:
- In the insured occupation we can observe most of the data is covered by machine inspector followed by professional speciality. Apart from this all the other insured occupations have almost same counts.
- With respect to insured hobbies, we can notice bungie-jumping covered the highest data followed by camping and paintball. And other categories have the average counts.
- With respect to the incident state, State5, State7 and State9 states have highest counts.
- In incident city, almost all the columns have equal counts.
- When we look at the vehicle manufactured companies, the categories Saab, Suburu, Dodge and Nissan have highest counts.
- Incase of DayOfWeek almost all they days have had equal number of incidents.
In [64]:
# Comparision between two variables
plt.figure(figsize=[18,13])
plt.subplot(2,2,1)
plt.title('Comparision CustomerLoyaltyPeriod and InsuredAge')
sns.scatterplot(x=train_df['CustomerLoyaltyPeriod'],y=train_df['InsuredAge'],hue=train_df['ReportedFraud'],palette="gnuplot");
plt.subplot(2,2,2)
plt.title('Comparision between AmountOfTotalClaim and AmountOfInjuryClaim')
sns.scatterplot(x=train_df['AmountOfTotalClaim'],y=train_df['AmountOfInjuryClaim'],hue=train_df['ReportedFraud'],palette="gnuplot");
plt.subplot(2,2,3)
plt.title('Comparision between AmountOfPropertyClaim and AmountOfVehicleDamage')
sns.scatterplot(x=train_df['AmountOfPropertyClaim'],y=train_df['AmountOfVehicleDamage'],hue=train_df['ReportedFraud'],palette="gnuplot");
plt.subplot(2,2,4)
plt.title('Comparision between CustomerLoyaltyPeriod and AmountOfTotalClaim')
sns.scatterplot(x=train_df['CustomerLoyaltyPeriod'],y=train_df['AmountOfTotalClaim'],hue=train_df['ReportedFraud'],palette="gnuplot");
From the above scatter plot we can observe the following things:
- There is a positive linear relation between InsuredAge and CustomerLoyaltyPeriod column. As InsuredAge increases the CustomerLoyaltyPeriod also increases, also the fraud reported is very less in hits case. In the second graph we can observe the positive linear relation, as total claim amount increases, injury claim is also increases. Third plot is also same as second one that is as the property claim increases, vehicle claim also increases. In the fourth plot we can observe the data is scattered and there is no much relation between the features.
In [65]:
fig,axes=plt.subplots(2,2,figsize=(12,10))
# Comparing insured_sex and age
sns.violinplot(x='InsuredGender',y='InsuredAge',ax=axes[0,0],data=train_df,palette="ch:.25",hue="ReportedFraud",split=True)
# Comparing policy_state and witnesses
sns.violinplot(x='InsurancePolicyState',y='Witnesses',ax=axes[0,1],data=train_df,hue="ReportedFraud",split=True,palette="hls")
# Comparing csl_per_accident and property_claim
sns.violinplot(x='CombinedSingleLimit',y='AmountOfPropertyClaim',ax=axes[1,0],data=train_df,hue="ReportedFraud",split=True,palette="Dark2")
# Comparing csl_per_person and age
sns.violinplot(x='SplitLimit',y='InsuredAge',ax=axes[1,1],data=train_df,hue="ReportedFraud",split=True,palette="mako")
plt.show()
Findings:
- The fraud report is high for both the males females having age between 30-45. The people who own the policy state "IN" have high fraud report. The person who has csl per accident insurance by claiming property in the range 5000-15000 have the fraud report. The SplitLimit with age 30-45 are facing the fraudulent reports.
In [66]:
fig,axes=plt.subplots(2,2,figsize=(12,12))
# Comparing insured_sex and age
sns.violinplot(x='ReportedFraud',y='AmountOfTotalClaim',ax=axes[0,0],data=train_df,hue="ReportedFraud" ,palette="hls")
# Comparing policy_state and witnesses
sns.violinplot(x='ReportedFraud',y='AmountOfVehicleDamage',ax=axes[0,1],data=train_df,hue="ReportedFraud",palette="cool_r")
# Comparing csl_per_accident and property_claim
sns.violinplot(x='ReportedFraud',y='AmountOfPropertyClaim',ax=axes[1,0],data=train_df,hue="ReportedFraud",palette="cividis")
# Comparing csl_per_person and age
sns.violinplot(x='ReportedFraud',y='AmountOfInjuryClaim',ax=axes[1,1],data=train_df,hue="ReportedFraud",palette="gnuplot2")
plt.show()
Findings:
- Most of the fraud reports found when the total claimed amount is 50000-70000. The fraud report is high when the claimed vehicle is between 37000-57000. The fraud reports is high when the property claimed is between 5200-8500. Most fraud reported when injury claim are between 5000 to 8000.
In [67]:
# Comparing policy_state and fraud_reported
sns.catplot(x='InsurancePolicyState',kind='count',data=train_df,hue='ReportedFraud',palette="Dark2")
plt.show()
Findings:
- Fraud report is bit high in the "State3" policy state.
In [68]:
# Comparing insured_education_level and fraud_reported
sns.catplot(x='InsuredEducationLevel',kind='count',data=train_df,hue='ReportedFraud',palette="tab20b_r")
plt.xticks(rotation=90)
plt.show()
Findings:
- The fraudulent level is very less for the people who have high school education and the people who have completed their "JD" education have high fraud report. The people who have high insured education are facing insurance fraudulent compared to the people with less insured education level.
In [69]:
# Comparing insured_occupation and fraud_reported
sns.catplot(x='InsuredOccupation',kind='count',data=train_df,hue='ReportedFraud',palette="spring_r")
plt.xticks(rotation=90)
plt.show()
Findings:
- The people who are in the position exec-managerials have high fraud reports compared to others.
In [70]:
# Comparing insured_hobbies and fraud_reported
sns.catplot(x='InsuredHobbies',kind='count',data=train_df,hue='ReportedFraud',palette="gnuplot2")
plt.xticks(rotation=90)
plt.show()
Findings:
- The fraud report is high for the people who have the hobby of playing chess and cross fit.
In [71]:
# Comparing insured_relationship and fraud_reported
sns.catplot(x='InsuredRelationship',kind='count',data=train_df,hue='ReportedFraud',palette="gist_earth")
plt.xticks(rotation=90)
plt.show()
Findings:
- The fraud report is high for the customers who have other relative and it is very less for husband and own-child.
In [72]:
# Comparing incident_type and fraud_reported
sns.catplot(x='TypeOfIncident',kind='count',data=train_df,hue='ReportedFraud',palette="Set2_r")
plt.xticks(rotation=90)
plt.show()
Findings:
- In Multivehicle collission and single vehicle collision, the fraud report is very high compared to others.
In [73]:
# Comparing collision_type and fraud_reported
sns.catplot(x='TypeOfCollission',kind='count',data=train_df,hue='ReportedFraud',palette="gist_earth")
plt.xticks(rotation=90)
plt.show()
Findings:
- The fraud level is high in the collision type Rear Collision and other two collision type have average reports.
In [74]:
# Comparing incident_severity and fraud_reported
sns.catplot(x='SeverityOfIncident',kind='count',data=train_df,hue='ReportedFraud',palette="mako")
plt.xticks(rotation=90)
plt.show()
Findings:
- The fraud report is high in Major damage incident severity and for Trivial Damage the report is less compared to others.
In [75]:
# Comparing authorities_contacted and fraud_reported
sns.catplot(x='AuthoritiesContacted',kind='count',data=train_df,hue='ReportedFraud',palette="magma")
plt.xticks(rotation=90)
plt.show()
Findings:
- The police contacted cases are very high and the fraud report is almost equal for all the authorities except None.
In [76]:
# Comparing incident_state and fraud_reported
sns.catplot(x='IncidentState',kind='count',data=train_df,col='ReportedFraud',palette="cubehelix")
plt.xticks(rotation=90)
plt.show()
Findings:
- The "State7" has high fraud reports compared to other states.
In [77]:
# Comparing incident_city and fraud_reported
sns.catplot(x='IncidentCity',kind='count',data=train_df,hue='ReportedFraud',palette="bright")
plt.xticks(rotation=90)
plt.show()
Findings:
- The "City1" and "City2" have very high fraud reports compared to others.
In [78]:
# Comparing property_damage and fraud_reported
sns.catplot(x='PropertyDamage',kind='count',data=train_df,col='ReportedFraud',palette="ocean")
plt.show()
Findings:
- The customers who have property damage case they have comparitively high fraud reports.
In [79]:
# Comparing police_report_available and fraud_reported
sns.catplot(x='PoliceReport',kind='count',data=train_df,col='ReportedFraud',palette="coolwarm")
plt.xticks(rotation=90)
plt.show()
Findings:
- If there is no police report available then the fraud report is very high.
In [80]:
# Comparing auto_make and fraud_reported
sns.catplot(x='VehicleMake',kind='count',data=train_df,hue='ReportedFraud',palette="bright")
plt.xticks(rotation=90)
plt.show()
Findings:
- In all the auto make cases the fraud report is almost same, except the "Ford" which is a bit on the higher end.
In [81]:
sns.pairplot(train_df,hue="ReportedFraud",palette="ocean")
plt.show()
